Apparatus and method for generating images

ABSTRACT

A low dynamic range (LDR) image data storage stores LDR image sets each of which includes a plurality of LDR segmented images captured with a camera panned. Here each LDR image set is of a different exposure level, respectively. A high-dynamic-range (HDR) synthesizing unit combines and merges a plurality of LDR segmented images of different exposure levels so as to generate an HDR segmented image set including a plurality of HDR segmented images of different pan angles or tilt angles. A panoramic image synthesizing unit generates an HDR panoramic image by stitching together adjacent HDR segmented images.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method forgenerating images.

2. Description of the Related Art

With the prevalence of digital still cameras and digital video cameras,there are increased occasions where still images or moving images havingbeen shot are stored in a computer for later viewing, processing, ordisplaying on the screen of a game device or a television system. It isalso popularly done that the shot moving images are uploaded to aposting site on the Internet so as to share them with the other users.

Among the digital cameras are those capable of shooting panoramicimages, which allow the image taking of panoramic images of wide viewangle with perfect ease. Also in wide use are software tools that cangenerate a panoramic image by stitching together a plurality of imagesshot by a digital camera from different shooting directions.

There is a site named “360cities” (http://www.360cities.net) thataccepts the posting of panoramic images shot by users and show them onthe Internet, so that the users around the world can view the panoramicimages posted.

A panoramic image is a spherical, or omnidirectional, image. When shotoutdoors, therefore, it may have the sun, street lamps, or such otherthings captured in it. As a result, the panoramic image may have anextremely wide dynamic range with great differences between brightportions and dark portions. Thus, if a panoramic image is shot withexposure adjusted to a specific object, then the resulting image maysometimes have “white-out” in the bright portions and “black-out” in thedark portions. As used herein, “white-out” is complete whitening ofportions exposed to strong light, whereas “black-out” is completeblackening of dark portions least exposed to light.

In a normal photo shoot, therefore, an object or objects are capturedwith correct exposure, or adjustments are made to prevent strong lightfrom entering a field of view, in order to avoid white-out or black-out.However, in the shooting of a panoramic image that covers alldirections, it may be out of the question to determine correct exposurematching a specific object or objects. To make up for this shortcoming,a panoramic image is, for instance, synthesized by the use of a highdynamic range synthesis technique. In this technique, a plurality of lowdynamic range photos are taken with exposure changed accordingly, andthen those photos are put together into an image having a high dynamicrange without white-out or black-out. Yet, this high dynamic rangesynthesis requires use of a high-priced camera or cameras and longshooting and processing time.

SUMMARY OF THE INVENTION

The present invention has been made in view of the foregoingcircumstances, and a purpose thereof is to provide a technology forefficiently generating images without white-out or black-out.

In order to resolve the above-described problems, an image generationapparatus according to one embodiment of the present invention includes:An image generation apparatus comprising: a storage configured to storea plurality of low-dynamic-range (LDR) image sets each including aplurality of low-dynamic-range (LDR) segmented images of different panangles or tilt angles, each LDR image set being of a different exposurelevel, respectively; a high-dynamic-range (HDR) synthesizing unitconfigured to combine a plurality of LDR segmented images of differentexposure levels so as to generate a high-dynamic-range (HDR) segmentedimage set including a plurality of high-dynamic-range (HDR) segmentedimages of different pan angles or tilt angles; and an output imagesynthesizing unit configured to output an HDR output image by stitchingtogether adjacent HDR segmented images.

Another embodiment of the present invention relates to a method forgenerating images. The method includes: reading out, by a processor, aplurality of low-dynamic-range (LDR) image sets from a storage devicefor storing a plurality of LDR image sets each including a plurality oflow-dynamic-range (LDR) segmented images of different pan angles or tiltangles, each LDR image set being of a different exposure level,respectively; combining, by a processor, a plurality of LDR segmentedimages of different exposure levels so as to generate ahigh-dynamic-range (HDR) segmented image set including a plurality ofHDR segmented images of different pan angles or tilt angles; andoutputting, by a processor, an HDR output image by stitching togetheradjacent high dynamic range segmented images.

Still another embodiment of the present invention relates to a programembedded in a non-transitory computer-readable medium. The programincludes: a reading module operative to read out a plurality oflow-dynamic-range (LDR) image sets from a storage device for storing aplurality of LDR image sets each including a plurality oflow-dynamic-range (LDR) segmented images of different pan angles or tiltangles, each LDR image set being of a different exposure level,respectively; a high dynamic range synthesizing module operative tocombine a plurality of LDR segmented images of different exposure levelsso as to generate a high-dynamic-range (HDR) segmented image setincluding a plurality of high-dynamic-range (HDR) segmented images ofdifferent pan angles or tilt angles; a tone mapping module operative tocompress dynamic range of the plurality of HDR segmented images ofdifferent pan angles or tilt angles by tone mapping so as to generate aplurality of compressed dynamic range segmented images of different panangles or tilt angles; and an output image synthesizing module operativeto generate an HDR output image by stitching together adjacent HDRsegmented images, when the HDR output image is to be generated, andoperative to generate an LDR output image by stitching together adjacentsegmented images out of the plurality of compressed dynamic rangesegmented images of different pan angles or tilt angles generated by thetone mapping, when the LDR output image is to be generated.

Optional combinations of the aforementioned constituting elements, andimplementations of the invention in the form of methods, apparatuses,systems, computer programs, data structures, recording media and soforth may also be effective as additional modes of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of examples only, withreference to the accompanying drawings which are meant to be exemplary,not limiting, and wherein like elements are numbered alike in severalFigures in which:

FIG. 1 is a configuration diagram of a panoramic image display apparatusaccording to an embodiment;

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice;

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system used toshoot panoramic images;

FIG. 4A is an illustration with which to explain azimuth angle θ of acamera;

FIG. 4B is an illustration with which to explain elevation angle φ of acamera;

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when an initial position of a camera is in a direction of azimuthangle θ;

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera is in a direction of elevation angle φ=60°;

FIG. 7A explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 7B explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 8 is a flowchart showing a procedure for generating a panoramicimage by the panoramic image display apparatus of FIG. 1;

FIG. 9A and FIG. 9B show control points detected between two adjacentsegmented images;

FIG. 10 shows how two adjacent segmented images are aligned and combinedusing control points;

FIGS. 11A and 11B are diagrams for explaining an extended panoramicimage having overlapped ends for a panoramic image;

FIG. 12 is an illustration for explaining a conventional technique forgenerating an HDR panoramic image from LDR segmented images for thepurpose of comparison;

FIG. 13 is an illustration for explaining a technique used in anembodiment for generating an HDR panoramic image from LDR segmentedimages; and

FIG. 14 is an illustration for explaining another technique of anembodiment for generating an LDR panoramic image from LDR segmentedimages.

DETAILED DESCRIPTION OF THE INVENTION

A description will be given of an outline of a preferred embodiment. Inthe present embodiments, a high dynamic range synthesis is performed onlow-dynamic-range (LDR) segmented images of different exposures. Thus,the segmented images having a high dynamic range is first generated andthen those segmented images are stitched together to generate apanoramic image.

FIG. 1 is a configuration diagram of a panoramic image display apparatus100 according to a preferred embodiment. For example, the panoramicimage display apparatus 100 as shown in FIG. 1 may be functionallyconfigured such that hardware, software or a combination of both isimplemented to a personal computer, a game device, a portable device, amobile terminal and so forth. Part of such functional components may beimplemented in a client, so that the panoramic image display apparatus100 may be realized as a server-client system via a network.

An HDR synthesizing unit 20, a tone mapping unit 21, a control pointdetector 22, a panoramic image synthesizing unit 23, an image codingunit 25, and an LDR image data storage 26 in the panoramic image displayapparatus 100 constitute a panoramic image generation apparatusaccording to an embodiment. These functional components of the panoramicimage generation apparatus may be implemented to a personal computer, agame device, a portable device, a mobile terminal and so forth otherthan the panoramic image display apparatus 100 and may be connected tothe panoramic image display apparatus 100 via a network or a connectioncable. Also, the functional components of the panoramic image generationapparatus may be implemented by use of a CPU or memory of a digitalcamera.

An LDR image data storage 26 stores an LDR image set that includes aplurality of low-dynamic-range (LDR) segmented images captured with acamera panned or tilted. Stored in the LDR image data storage 26 are aplurality of LDR image sets of different exposures. A panoramic image iscreated by stitching together the segmented images of different panangles and tilt angles. For easier synthesis, however, the images areshot with overlap in the joint regions of adjacent segmented images. TheLDR image set may also be one shot with a plurality of camerasassociated with different pan angles or tilt angles for the shooting. Itshould be noted that the segmented images whether they are shot with asingle camera panned or tilted or with a plurality of cameras associatedwith different pan angles or tilt angles for the shooting are referredto herein simply as “segmented images of different pan angles or tiltangles”.

The HDR synthesizing unit 20 generates a plurality of high-dynamic-range(HDR) image sets of different pan angles or tilt angles as follows. Thatis, those HDR segmented images are generated by reading out a pluralityof LDR image sets of different exposures from the LDR image data storage26 and synthesizing a plurality of low-dynamic-range segmented images ofdifferent exposures for each of the combinations of pan angle and tiltangle at which the segmented images have been shot.

The tone mapping unit 21, which is configured to operate when apanoramic image of low dynamic range is to be generated, does notoperate when a panoramic image of high dynamic range is to be generated.

A description will first be given of a case where a panoramic image ofhigh dynamic range is to be generated.

The HDR synthesizing unit 20 supplies a plurality of generatedhigh-dynamic-range segmented images of different pan angles or tiltangles to the control point detector 22. The control point detector 22detects control points for associating adjacent high-dynamic-rangesegmented images with each other by extracting feature points betweenthe adjacent segmented images out of the plurality of high-dynamic-rangesegmented images of different pan angles or tilt angles.

A technology known in the art for image matching may be used in aprocess of feature point extraction. Upon completion of the processingfor feature point extraction, control points best suited for theassociation of adjacent segmented images are selected from among theextracted feature points. While there are a multiplicity of featurepoints that are extracted, it is desirable that the control points bescattered evenly over the entire joint regions of adjacent segmentedimages. The control points scattered evenly over the entire jointregions will accomplish alignment of adjacent segmented images with highaccuracy.

The panoramic image synthesizing unit 23 synthesizes ahigh-dynamic-range panoramic image by adjusting the alignment of theadjacent high-dynamic-range segmented images using the detected controlpoints, thereby generating a panoramic image of high dynamic range. Inother words, the synthesis is carried out by rotating segmented imagesin such a manner as to achieve a maximum agreement between thecorresponding control points of adjacent segmented images, that is, tominimize the total of gaps between the corresponding control points ofadjacent segmented images.

Next, a description will be given of a case where a panoramic image oflow dynamic range is to be generated.

When a panoramic image of low dynamic range is to be generated, the tonemapping unit 21 is operated. The HDR synthesizing unit 20 supplies aplurality of generated high-dynamic-range segmented images of differentpan angles or tilt angles to the tone mapping unit 21. The tone mappingunit 21 generates a plurality of compressed dynamic range segmentedimages of different pan angles or tilt angles by compressing the dynamicrange of the plurality of high-dynamic-range segmented images ofdifferent pan angles or tilt angles by tone mapping.

The control point detector 22 detects control points for associatingadjacent compressed dynamic range segmented images with each other byextracting feature points between the adjacent segmented images out ofthe plurality of compressed dynamic range segmented images of differentpan angles or tilt angles.

The panoramic image synthesizing unit 23 synthesizes a low-dynamic-rangepanoramic image by adjusting the alignment of the adjacent compresseddynamic range segmented images using the detected control points,thereby generating a panoramic image of low dynamic range.

The image coding unit 25 codes the panoramic image of high dynamic rangeor low dynamic range combined by the panoramic image synthesizing unit23. In so doing, an extended panoramic image is generated such that partof a region at a right end of the panoramic image is added to a left endthereof and such that part of a region at the left end thereof is addedto the right end thereof. And the thus generated extended panoramicimage is compressed and coded. If the panoramic image is a moving imageor moving images, the image coding unit 25 will performmotion-compensated prediction on the extended panoramic image on amacroblock-by-macroblock basis and then compress and code the thusprocessed extended panoramic image. The image coding unit 25 stores thecoded panoramic image in a panoramic image/additional data storage 24.

The panoramic image/additional data storage 24 stores panoramic imageshaving information on shooting locations and information on shootingorientations associated with each other. The additional data, such asinformation on shooting locations and shooting orientations, may beadded directly to a data file of panoramic images, or may be managed asa separate file from the panoramic images.

The information on shooting locations includes, for instance,information on latitudes and longitudes which is given by GPS (GlobalPositioning System). The information on shooting orientation includes,for instance, information on the azimuth (angle of orientation) of thecenter point of a panoramic image obtained from an azimuth sensor, andmay also additionally include information on the elevation angle androll angle of a camera at the time of shooting.

If the azimuth of the center point of a panoramic image is given asinformation on a shooting orientation, then it is possible to calculatethe orientation of an arbitrary point of the panoramic image based onthe angle of panning the camera to the right or left. The panoramicimages may have, as the information on the shooting orientations, thecoordinate values of pixels in the orientations of true north, truesouth, true east, and true west of the panoramic images which arecalculated based on the azimuths and pan angles of the center points ofthe panoramic images.

A panoramic image acquiring unit 10 acquires a panoramic image to bedisplayed from the panoramic image/additional data storage 24. Thepanoramic image to be displayed is identified as the user specifies ashooting location on a map or the like.

An image decoding unit 12 decodes the extended panoramic image acquiredby the panoramic image acquiring unit 10, then trims the overlappedregions at both ends of the extended panoramic image, and supplies thereproduced panoramic image to the mapping processing unit 14. If theextended panoramic image acquired by the panoramic image acquiring unit10 is a moving image or moving images coded by the motion-compensatedprediction, the image decoding unit 12 will trim the overlapped regionsat the both ends thereof and supplies a frame or frames of thereproduced moving image(s) to a mapping processing unit 14.

The mapping processing unit 14 processes a mapping of a panoramic imageonto a three-dimensional panoramic space as textures.

In the case of a spherical, or omnidirectional (celestial), panoramicimage, a sphere is assumed as a three-dimensional panoramic space, andthe panoramic image is texture-mapped onto the spherical surface by asphere mapping. Or a cube may be assumed as a three-dimensionalpanoramic space, and the panoramic image may be texture-mapped onto thecubic surface by a cube mapping. Also, in the case where the panoramicimage does not have any component in tilt directions and spreads only inthe panning directions, a cylinder may be assumed as a three-dimensionalpanoramic space, and the panoramic image may be texture-mapped onto thecylindrical surface by a texture mapping. The same applies to the casewhere the panoramic image does not have any component in the panningdirections and spreads only in tilt directions.

A 3D image generator 16 generates a three-dimensional (3D) panoramicimage when the 3D panoramic space having a panoramic imagetexture-mapped thereon by the mapping processing unit 14 is viewed in aspecified line of sight. When the 3D panoramic space is a sphere, theviewpoint is placed at the center of the sphere. When the 3D panoramicspace is a cube, the viewpoint is placed at the center of the interiorof the cube. And when the 3D panoramic space is a cylinder, theviewpoint is placed on the center axis of the cylinder. The viewpoint isthe location where the panoramic image to be displayed is shot, and theline of sight is the direction in which the surrounding area is viewedand is thus identified by the azimuth and the elevation angle. The 3Dimage generator 16 generates a 3D image when the 3D panoramic space isviewed in the line of sight identified by the azimuth and the elevationangle.

A display control unit 18 has a 3D panoramic image or a map image thusgenerated displayed on a screen of the display unit.

A user interface 40 is a graphical user interface through which the usercan manipulate the graphics displayed on the screen of a display usingan input device. The user interface 40 receives user instructions on themap or 3D panoramic image displayed on the screen from the input devicewhich may be a controller of a game device, a mouse, a keyboard, or thelike. FIG. 2 shows a controller 102 as an example of the input device,whose construction will be discussed in detail later.

The user interface 40 instructs the panoramic image acquiring unit 10 toacquire a specified panoramic image from the panoramic image/additionaldata storage 24.

The user can input instructions to change the line of sight for viewingthe 3D panoramic space by operating an analog stick 118 or directionkeys 116 of the controller 102, for instance. A line-of-sight settingunit 32 of the user interface 40 gives a line of sight instructed by theuser to the 3D image generator 16. The 3D image generator 16 generatesan image when the 3D panoramic space is viewed in a specified line ofsight.

An angle-of-view setting unit 31 sets an angle of view when the user hasperformed a zoom operation on the panoramic image being displayed andgives the information of the angle of view thus set to the panoramicimage acquiring unit 10 and the 3D image generator 16. Where panoramicimages of different angles of view are stored in the panoramicimage/additional data storage 24, the panoramic image acquiring unit 10reads out a panoramic image of an angle of view closest to the set angleof view and changes the panoramic image to be displayed. The 3D imagegenerator 16 realizes the visual effects of zoom-in and zoom-out byenlarging or reducing the 3D panoramic image according to the set angleof view.

A panoramic image may have information on the shooting altitude, and thepanoramic image/additional data storage 24 may store panoramic imagesshot at different altitudes at the same shooting location. In such acase, the user can input instructions to change the altitude byoperating L1/L2 buttons 161 and 162 provided on the left front of thecasing of the controller 102, for instance. Pressing the L1 button 161will give an instruction to raise the altitude, and pressing the L2button 162 will give an instruction to lower the altitude.

The display control unit 18 may indicate to the user, for instance, withsmall arrows at the top and bottom portions of the screen that thepanoramic image currently being displayed has panoramic images shot atdifferent altitudes at the same shooting location. An arrow facingupward in the top portion of the screen indicates the presence of apanoramic image shot at a higher altitude than the current one, and anarrow facing downward in the bottom portion of the screen indicates thepresence of a panoramic image shot at a lower altitude than the currentone.

Upon receipt of an instruction from the user to change the altitude, thealtitude setting unit 34 of the user interface 40 instructs thepanoramic image acquiring unit 10 to acquire a panoramic imagecorresponding to the specified altitude, despite the same latitude andlongitude, from the panoramic image/additional data storage 24. Thepanoramic image acquiring unit 10 acquires a panoramic image of a highershooting altitude than the panoramic image currently being displayedwhen the L1 button 161 is pressed, and acquires a panoramic image of alower shooting altitude than the current one when the L2 button 162 ispressed.

When a display is produced by switching to a panoramic image of adifferent shooting altitude, the display control unit 18 may give aspecial effect to the image so that the user may have a sense of ridingan elevator up or down. For example, when switching to a panoramic imageof a higher altitude, the panoramic image currently being displayed canbe scrolled downward, thereby having the panoramic image of a higheraltitude descend from above with the result that the user may have asense of having risen upstairs.

A panoramic image contains information on the shooting date and time,and the panoramic image/additional data storage 24 may store panoramicimages shot at different dates and times at the same shooting location.In such a case, the user can input instructions to change the date andtime by operating R1/R2 buttons 151 and 152 provided on the right frontof the casing of the controller 102, for instance. Pressing the R1button 151 will give an instruction to shift to a later date and time,and pressing the R2 button 152 will give an instruction to shift to anearlier date and time.

The display control unit 18 may indicate to the user, for instance, withwatch and calendar icons in the corner of the screen that the panoramicimage currently being displayed has panoramic images shot at differentdates and times. Watch icons may be displayed to indicate the presenceof panoramic images for different times of day such as morning, noon,and night, where a calendar of icons may be displayed to indicate thepresence of panoramic images for different seasons such as spring,summer, autumn, and winter.

Upon receipt of an instruction from the user to change the date andtime, the date/time setting unit 36 of the user interface 40 instructsthe panoramic image acquiring unit 10 to acquire a panoramic imagecorresponding to a specified date and time at the same shooting locationfrom the panoramic image/additional data storage 24. The panoramic imageacquiring unit 10 acquires a panoramic image of a later shooting dateand time than the panoramic image currently being displayed when the R1button 151 is pressed, and acquires a panoramic image of an earliershooting date and time than the current one when the R2 button 152 ispressed.

Thus, it is possible to switch the panoramic image being displayed topanoramic images of a different time of day or season at the sameshooting location, for example, from one shot in the morning to one shotat night, or from one shot in spring to one shot in autumn. In changingthe panoramic image, the display control unit 18 may give an effect offade-in and fade-out to the image.

A viewpoint position setting unit 30 set the shooting location of apanoramic image as a viewpoint position and conveys it to the 3D imagegenerator 16. The line-of-sight setting unit 32 sends the specifiedline-of-sight to the 3D image generator 16.

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice. The panoramic image display apparatus 100 may be a game device,for instance.

The controller 102 has a plurality of buttons and keys to receivecontrol inputs to the panoramic image display apparatus 100. As the useroperates on the buttons or keys of the controller 102, their operationinputs are transmitted to the panoramic image display apparatus 10through wireless or wired connections.

Provided on a casing top surface 122 of the controller 102 are a groupof arrow keys 116, analog sticks 118, and a group of operation buttons120. The group of direction keys 116 include “up-”, “down-”, “left-”,and “right-” direction indication keys. The group of operation buttons120 include a circle button 124, a cross button 126, a square button128, and a triangle button 130.

The user holds a left-hand grip 134 b with the left hand and holds aright-hand grip 134 a with the right hand, and operates the group ofdirection keys 116, the analog sticks 118, and the group of operationbuttons 120 on the casing top surface 122.

Provided on a front side of the controller 102 are a right-handoperation part 150 and a left-hand operation part 160. The right-handoperation part 150 includes an R1 button and an R2 button, whereas theleft-hand operation part 160 includes an L1 button 161 and an L2 button162.

The user can shift a pointer displayed on the screen in vertical andhorizontal directions by operating the directional key group 116. Forexample, when selecting one of a plurality of markers displayed within apanoramic image, the user can shift the pointer between the plurality ofmarkers on the screen by operating the directional key group 116. Theuser can select a desired marker by pressing the circle button 124 whenthe pointer has come upon the marker.

Different functions may be assigned to the respective buttons ofoperation buttons 120 by a panoramic image display application program.For example, the function to specify the display of a menu is assignedto the triangle button 130, the function to specify the cancel of aselected item is assigned to the cross button 126, the function tospecify the determination of a selected item is assigned to the circlebutton, and the function to specify the display/non-display of table ofcontents or the like is assigned to the square button 128.

The analog sticks 118 have means to output analog values as they aretipped by the user. The controller 102 sends an analog output signalcorresponding to the direction and amount of tipping of the analog stick118 to the panoramic image display apparatus 100. For example, the usercan shift the viewpoint in a desired direction within a 3D panoramicimage shown on the display by tipping the analog stick 118 in thedesired direction.

The casing top surface 122 is further provided with an LED button 136, aselector button 140, and a start button 138. The LED button 136 is usedas the button for the display of the menu screen on the display, forinstance. The start button 138 is the button with which the userinstructs the start of a panoramic image display application, the startor pause of playback of a panoramic image, or the like. The selectorbutton 140 is the button with which the user instructs a selection froma menu display shown on the display or the like.

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system 230 usedto shoot panoramic images.

As shown in FIG. 3D, a camera 200 in the omnidirectional image shootingsystem 230 is secured onto a control disk 210. And a camera's pan anglecan be changed as the control disk 210 is rotated around a Z axis, acamera's tilt angle can be changed as the control disk 210 is rotatedaround an X axis, and a camera's roll angle can be changed as thecontrol disk 210 is rotated around a Y axis. The Z axis herein is thevertical axis (gravitational direction axis).

FIG. 3A is a top view of the camera 200 installed on the control disk210. The initial position (Y-axis direction) of the control disk is panangle 0°, and the pan angle can be changed within a range of −180° to+180° around the Z axis.

FIG. 3B is a front view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is roll angle 0°, andthe roll angle can be changed within a range of −180° to +180° aroundthe Y axis.

FIG. 3C is a side view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is tilt angle 0°, andthe tilt angle can be changed within a range of −90° to +90° around theX axis.

In order to endow a panoramic image shot by the omnidirectional imageshooting system 230 of FIG. 3D with information on the shootingorientations, it is necessary to record the orientations of the camera200 at the time of image taking. For that purpose, the omnidirectionalimage shooting system 230 is provided with an azimuth sensor formeasuring orientations and an acceleration sensor for measuring tiltangles. The omnidirectional image shooting system 230 is furtherprovided with a GPS sensor or the like for measuring the shootinglocation and time.

FIG. 4A is an illustration with which to explain azimuth angle θ of thecamera 200, and FIG. 4B is an illustration with which to explainelevation angle φ of the camera 200. FIG. 4A is a top view of the camera200, in which the camera 200 in an initial position of shooting faces adirection 220 which is azimuth angle θ displaced from true north toeast. This direction is equal to pan angle 0°. In other words, theazimuth angle of the reference direction 220 of the pan angle is θ. Whenshooting a panoramic image, the image of an object is takenpanoramically by changing the pan angle in a range of −180° to +180°with respect to the reference direction 220 of the azimuth angle θ.

FIG. 4B is a side view of the camera 200. The elevation angle φ is thedirection of tilt 0°, which is an angle where an upper direction isdefined to be positive in relation to the Y-axis direction, when thecamera 200 is rotated around the X axis. Normally, the elevation angle φis 0° since the image taking is done with the camera 200 set in ahorizontal position. To shoot a spherical panoramic image, however, itis necessary to take the images of the object by changing the elevationangle φ with the tilt of the camera.

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when the initial position of the camera 200 is in a direction ofthe azimuth angle θ.

As shown in the top view of FIG. 5A, the camera 200 in the initialposition faces the direction 220 of azimuth angle θ. And as shown in theside view of FIG. 5B, the elevation angle of the camera 200 is φ=0°.With the elevation angle kept at φ=0°, an omnidirectional panoramic viewis shot at the elevation angle φ=0° while the pan angle of the camera200 with respect to the reference direction 220 is varied within a rangeof −180° to +180°. FIG. 5C is a panoramic image 300 taken in theabove-described manner. At the center of the panoramic image 300, thepan angle is 0°. The left half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to−180°. Similarly, the right half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to180°.

The central position of the pan angle 0° is displaced from true north byazimuth angle θ. Thus, the positions of north (N), south (S), east (E),and west (W) are those indicated by dotted lines. As long as thepanoramic image 300 contains the azimuth angle θ of the central positionof pan angle 0° as the information on the shooting orientations, thepixel positions of north (N), south (S), east (E), and west (W) can beevaluated in consideration of a displacement of the azimuth angle θ.Alternatively, instead of the azimuth angle θ, the coordinate values ofpixel positions of north (N), south (S), east (E), and west (W) may beused as the information on the shooting orientations.

In order to obtain a spherical panoramic image, it is necessary to takeimages by varying the elevation angle of the camera 200. For example, ifthe angle of view of the camera 200 is 60°, a spherical panoramic imagecan be theoretically obtained as follows. That is, the camera 200 istilted vertically at ±60®, and the similar image taking is done byvarying the pan angle within a range of −180° to +180°.

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera 200 is in a direction of elevation angle φ=60°. Asshown in the top view of FIG. 6A, the camera 200 in the initial positionfaces the direction 220 of azimuth angle θ. And as shown in the sideview of FIG. 6B, the elevation angle of the camera 200 is φ=0°. With theelevation angle kept at φ=60°, a panoramic view 302 as shown in FIG. 6Cis shot at the elevation angle φ=60° while the pan angle of the camera220 with respect to the reference direction 220 is varied within a rangeof −180° to +180°.

With the elevation angle kept at φ=−60°, a panoramic view 302 issimilarly shot at the elevation angle θ=−60° while the pan angle isvaried within a range of −180° to +180°. A spherical panoramic image isobtained by combining the panoramic images shot at the elevation anglesφ=0°, 60°, and −60°. However, in implementation, a method is oftenemployed where the vicinities of a boundary (bordering areas) are takenin an overlapped manner, in order to correct the mismatch caused by lensdistortions when images are stitched together in boundary portions atthe angle of view.

The spherical panoramic image obtained as described above is endowedwith information on azimuth angles and elevation angles. Therefore, itis possible to identify the azimuth and elevation angle of an arbitrarypixel of the panoramic image based on the information. Also, thepanoramic image is provided with the latitude and longitude informationmeasured by GPS as the positional information of the shooting location.The additional information to be attached to the panoramic image may berecorded, for example, in the format of image file called Exif(Exchangeable Image File Format). The place-name of the shootinglocation can be recorded in a part of the file name, whereas theshooting date and time, the latitude and longitude of the shootinglocation, the altitude, the azimuth angle, and the like can be recordedas data in the Exif format. The elevation angle, which is not defined inthe Exif format, is recorded as extended data.

FIG. 7A and FIG. 7B are illustrations with which to explain a method ofcreating a panoramic image by stitching a plurality of images together.

In the example of FIG. 7A, seven images 341 to 347 shot by tilting (orpanning) the camera 200 are mapped into a cylinder and then stitchedtogether to prepare a cylindrical image 340. When the images arestitched together, the bordering areas of the images are overlapped witheach other.

As shown in FIG. 7B, a plurality of cylindrical images like one shown inFIG. 7A are obtained in the panning (or tilting) direction by theshooting with the panning (or tilting) of the camera 200. Anomnidirectional panoramic image 360 is finally obtained by synthesizingthese cylindrical images 340 a to 340 f with the bordering areas of theimages overlapped.

FIG. 8 is a flowchart showing a procedure for generating a panoramicimage by the panoramic image display apparatus 100. With reference toFIGS. 9A and 9B to FIGS. 11A and 11B, each step in the procedure forgenerating a panoramic image of FIG. 8. In the flowchart shown in FIG.8, the procedure of each structural component is shown using S (thecapital letter of “Step”), which means a step, and numbers combined. Ifa determining process is executed in a processing indicated by thecombination of S and a number and if the decision result is positive,“Y” (the capital letter of “Yes”) will be appended like “(Y of S14)”.If, on the other hand, the decision result is negative, “N” (the capitalletter of “No”) will be appended like “(N of S14)”.

A plurality of LDR image sets of different exposures are inputted (S10).Each LDR image set includes a plurality of low dynamic rage segmentedimages that differ in at least one of pan angle and tilt angle.

The HDR synthesizing unit 20 combines or synthesizes a plurality of LDRsegmented images of different exposure for each of the combinations ofpan angle and tilt angle at which the segmented images have been shot,and thereby generates a plurality of HDR segmented images of differentpan angles or tilt angles (S12). A technology known in the art may beused in this high dynamic range synthesis technique. Regions where thesegments obviously having black-out and white-out are excluded from aplurality of LDR images of different exposures are selected andsynthesized. Thus, HDR images free from white-out and black out over theentire regions are generated.

If the output image format is of an LDR panoramic image (Y of S14), atone mapping process of Step S16 will be carried out. The tone mappingunit 21 performs the tone mapping process on the HDR segmented imagesgenerated by the HDR synthesizing unit 20, compresses the dynamic range,and thereby converts the HDR segmented images into the compresseddynamic range segmented images. A technique of the tone mapping may be amethod for reducing the contrast of the entire image and a method wherethe images are converted non-linearly by mimicking the human's visualfeatures, for instance. With the tone mapping process of Step S16completed, the procedure advances to Step S18.

If the output image format is of an HDR panoramic image (N of S14), thetone mapping process of Step S16 will be skipped and the procedure willadvance to Step S18.

The control point detector 22 extracts control points between adjacentsegmented images out of a plurality of compressed dynamic rangesegmented images/HDR segmented images of different pan angles or tiltangles, and thereby detects control points for associating adjacentsegmented images with each other (S18).

FIG. 9A and FIG. 9B show control points detected between two adjacentsegmented images 401 and 402. In FIG. 9A and FIG. 9B, nine controlpoints are detected.

The panoramic synthesizing unit 23 generates LDR/HDR panoramic images byadjusting the alignment of and synthesizing the adjacent compresseddynamic range segmented images based on the detected control points(S20).

FIG. 10 shows how the two adjacent segmented images 401 and 402 arealigned and combined using control points. The segmented image 402 isrotated relative to the other segmented image 401 in such a manner thatthe control points of the segmented image 401 and those of the segmentedimage 402 can achieve a maximum agreement between the correspondingpoints (indicated by circles in FIG. 10) in their overlapped region

The image coding unit 25 codes the LDR/HDR panoramic images obtained byadjusting the alignment of and synthesizing the images (S22). At thistime, part of a region at a right end of the panoramic image is added toa left end thereof, and part of a region at the left end thereof isadded to the right end thereof. And the thus generated extendedpanoramic image is compressed and coded, so that the block noise isreduced. If the panoramic image is a moving image or moving images, theimage coding unit 25 will perform the macroblock-by-macroblockmotion-compensated prediction on the extended panoramic image and codethe thus processed extended panoramic image. As a result,motion-compensated prediction that suppresses the block noise can beperformed even though there is a motion across the border lines at theboth ends of the panoramic image.

FIGS. 11A and 11B are diagrams for explaining an extended panoramicimage having overlapped ends for a panoramic image. FIG. 11A representsa panoramic image 500 shot with the camera panned 360 degrees. Thepanoramic image 500, when it is a moving image, is subjected to apredictive coding by referencing the pixels in peripheral regions usingmacroblock-by-macroblock motion vectors. Yet, the panoramic image 500whose right end and left end have no peripheral pixels on their rightand left, respectively, is bound to produce block noise. Thus, when thepanoramic image 500 is pasted to a sphere, unnatural border lines mayappear at the joints during reproduction of moving images.

To solve this problem, the panoramic image 500 is extended by copying aright end region and a left end region thereof at the left end and theright end thereof, respectively. FIG. 11B shows an extended panoramicimage 510. If the right end region 502 and the left end region 504 areeach a region for a pan angle of 20 degrees, then the extended panoramicimage 510 will be in an extended form for the pan angle of 400 degrees.Now the macroblock-by-macroblock motion-compensated predictive coding isperformed on this extended panoramic image 510, and the block noise tooccur at the right end and the left end of the panoramic image 500 willbe minimized.

With the motion-compensated predictive coding on the extended panoramicimage 510 of FIG. 11B completed, the copied right end region 502 andleft end region 504 are deleted to regain the panoramic image 500 ofFIG. 11A. After that, when the panoramic image 500 is pasted to thesphere (celestial sphere), border lines may not appear at the jointsduring reproduction of moving images because block noise at the jointsis minimized.

Next, with reference to FIG. 12 to FIG. 14, a detailed description willbe given of the advantages of the techniques of the present embodimentin generating an HDR/LDR panoramic image from a plurality of LDRsegmented images of different pan angles or tilt angles.

FIG. 12 is an illustration for explaining a conventional technique forgenerating an HDR panoramic image from LDR segmented images for thepurpose of comparison.

In this example, there are three sets of a plurality of LDR segmentedimages of different pan angles or tilt angles, with the exposure of eachset differing from each other. The exposure values of LDR segmentedimage sets 411, 412, and 413 are EV1, EV2, and EV3, respectively, thevalues descending in this order. Each of the LDR segmented image sets411, 412, and 413 includes six LDR segmented images of different panangles or tilt angles. For example, the six LDR segmented images areshot with the pan angle changed in three stages (p1, p2, p3) and thetilt angle changed in two stages (t1, t2).

Although three levels of exposure value are used here for convenience ofexplanation, the number N of exposure levels must be about 8 or more inactual implementations. Also, the number P of shots in the panningdirection is given by P=360/(horizontal angle of view×0.8), whereas thenumber T of shots in the tilting direction is given by T=180/(verticalangle of view×0.8). Notice that there may be cases where one each ofzenith and nadir shots is added to these in the tilting direction.Multiplication by the factor 0.8 is to provide an overlapped regionbetween adjacent segmented images. The total number M of segmented imageshots can be given by N×P×T.

Using the conventional technique, an LDR panoramic image 421 isgenerated by synthesizing the LDR segmented image set 411 of theexposure value EV1. Similarly, an LDR panoramic image 422 and an LDRpanoramic image 423 are generated by synthesizing the LDR segmentedimage set 412 of the exposure value EV2 and the LDR segmented image set413 of the exposure value EV3, respectively. When a panoramic image isgenerated by synthesizing segmented images, matching is performed bydetecting control points in the overlapped region of adjacent segmentedimages.

In doing so, there may possibly be some white-outs or black-outs withinan LDR segmented image set shot at a certain exposure value because ofimproper exposure of some segmented images. This is because there may beportions having been exposed to strong light or portions extremely darkin some of the segmented images shot with the pan angle or the tiltangle changed. Since it is difficult to extract feature points from thesegmented images having white-outs or black-outs, the control pointscannot be set, or even if they can be set, the control points may belocated unevenly or fewer in number. As a result, it is not possible tostitch together adjacent segmented images with perfection, thus endingup in an inferior synthesis of a panoramic image.

Next, an HDR panoramic image 430 is generated by a high dynamic rangesynthesis of the LDR panoramic image 421 of the exposure value EV1, theLDR panoramic image 422 of the exposure value EV2, and the LDR panoramicimage 423 of the exposure value EV3. However, as mentioned above, theLDR panoramic images 421, 422, and 423 may possibly include ones ofinferior synthesis due to white-outs or black-outs. Therefore, the HDRpanoramic image 430 finally synthesized may turn out inferior inaccuracy.

FIG. 13 is an illustration for explaining a technique of the presentembodiment for generating an HDR panoramic image from LDR segmentedimages.

In the present embodiment, an HDR panoramic image is generated by firstgenerating a set of HDR segmented images by a high dynamic rangesynthesis on the sets of LDR segmented images of different exposurevalues and then synthesizing them into a panoramic image.

An HDR panoramic image set 440 is generated by a high dynamic rangesynthesis of the LDR segmented image sets 411, 412, and 413 of theexposure values EV1, EV2, and EV3, respectively. The HDR segmented imageset 440 includes six HDR segmented images of different pan angles ortilt angles. Next, an HDR panoramic image 450 is generated bysynthesizing the set of HDR panoramic images 440.

According to this method, for an LDR segmented image of pan angle p1 andtilt angle t1, for instance, an HDR segmented image of pan angle p1 andtilt angle t1 is generated by performing a high dynamic range synthesison LDR segmented images of three exposure values EV1, EV2, and EV3.Accordingly, when the HDR segmented image is generated, the problem ofwhite-outs or black-outs having occurred with the LDR segmented imagescan be eliminated. When the HDR panoramic image 450 is generated bysynthesizing the set of HDR segmented images 440, the six HDR segmentedimages are free from white-outs or black-outs. Therefore, control pointscan be detected with high accuracy by reliably extracting feature pointsfrom the overlapped region of adjacent segmented images. The HDRpanoramic image 450 thus synthesized will turn out superior.

FIG. 14 is an illustration for explaining another technique of thepresent embodiment for generating an LDR panoramic image from LDRsegmented images. It differs from the technique of FIG. 13 in that atone mapping is performed on the HDR segmented image set 440, which hasbeen obtained through a high dynamic range synthesis on the LDRsegmented image sets 411, 412, and 413 of the exposure values EV1, EV2,and EV3, respectively, and an LDR panoramic image 470 is synthesizedafter this conversion of the HDR segmented image set 440 into an LDRsegmented image set 460. In this case, too, when the HDR segmented imageset 440 is generated, the problem of white-outs or black-outs has beeneliminated. Therefore, when the LDR segmented image set 460 with lowereddynamic range is to be synthesized, the problem of inability to detectcontrol points will not occur.

As described above, the techniques of the present embodiments as shownin FIG. 13 and FIG. 14 display markedly improved accuracy of thepanoramic image over that of the conventional technique of FIG. 12. Themethods of generating panoramic images according to the presentembodiments have advantages in processing efficiency and other aspects.

The conventional technique requires more processing time because the LDRpanoramic image is generated by detecting control points for the LDRsegmented image sets of their respective exposure values. In contrast tothis, the techniques of the present embodiments generate the HDRpanoramic image by detecting control points after the generation of anHDR segmented image set in advance. Accordingly, the processing time isreduced to 1/N of that of the conventional technique when N is thenumber of exposure levels.

Also, the conventional technique generates the LDR panoramic image froma plurality of LDR segmented images for each exposure value. Therefore,it requires the total number M of shots (M=number N of exposurelevels×number P of pan stages×number T of tilt stages), thus taking moreshooting time. Moreover, because of the longer shooting time, anymovement of objects may result in the shifting of the objects betweenLDR panoramic images of different exposure values. As a result, the HDRpanoramic image, when synthesized, may present an image containing blursof the objects.

In contrast to this, the techniques of the present embodiments have noneed for images of exposure values that are prone to cause white-outs orblack-outs. Therefore, without the shooting at such exposure values, HDRsegmented images of sufficient accuracy can be generated. The imagetaking may be done by previously determining proper exposure values inthe respective shooting directions by scanning the whole sphere ordetermining proper exposure values in the respective shooting directionsbased on the direction of light. That is, there is no need for shootingfor all the exposure values. Thus the shooting time can be shortened,and the processing time for HDR synthesis can be shortened also.

Moreover, the capacity of the techniques of the present embodiments toshorten the shooting time can reduce the possibilities of capturingfast-moving objects such as birds and airplanes. Even when there are LDRsegmented images having captured such moving objects, those images canbe deleted. With the presence of LDR segmented images for the same areaat other exposure values, an HDR segmented image with sufficientaccuracy can be produced.

As described above, the panoramic image display apparatus according tothe present embodiments can generate HDR/LDR panoramic images from LDRsegmented images efficiently with high accuracy. In the past, theshooting of panoramic images in low dynamic range has tended to produceinferior panoramic images due to the inability to shootomnidirectionally at proper exposure particularly when the front isbright against dark background. According to the techniques ofgenerating panoramic images of the present embodiments, HDR segmentedimages are first generated from LDR segmented images shot with theexposure varied before the panoramic image is synthesized. Thus superiorpanoramic images adjusted omnidirectionally for proper exposure valuescan be obtained.

The problem of white-outs or black-outs unavoidable with the LDRsegmented images is solved by this high dynamic range processing. Hence,a panoramic image can also be synthesized from HDR segmented images withhigh accuracy. Also, the shooting time can be markedly shortened byrefraining from the shooting at exposure values that are prone to causewhite-outs or black-outs. Moreover, even when there are unexpectedly LDRsegmented images having captured some fast-moving object, such an imagecan be deleted and an HDR segmented image can be generated. Thisprovides increased resistance to noise.

The present invention has been described based upon illustrativeembodiments. These embodiments are intended to be illustrative only andit will be obvious to those skilled in the art that variousmodifications to the combination of constituting elements and processescould be developed and that such modifications are also within the scopeof the present invention.

The description thus far has assumed the mapping of a panoramic image ona 3D panoramic space such as a spherical surface and the display on thescreen of a 3D panoramic image as the 3D panoramic space is viewed alongthe specified line of sight. However, the panoramic image may bedisplayed simply two-dimensionally. Then there is no need for themapping processor 14 and the 3D image generator 16 as part of theconstitution. This will simplify the panoramic image display apparatus100.

Panoramic images herein are not limited to ones shot by theomnidirectional image shooting system as shown in FIG. 3, but they maybe ones shot through a fish-eye lens or ones synthesized from aplurality of images shot with a normal digital camera in differentshooting directions.

Components, which are mainly involved in the processings of HDRsynthesis, tone mapping, detection of control points, panoramic imagesynthesis, and image coding, in the functional components of thepanoramic image display apparatus 100 may be implemented in a server.Also, components, which are mainly involved in the decoding andreproduction of panoramic image may be implemented in a client. Hence,the panoramic image display apparatus 100 may be realized as aserver-client system via a network. The server may generate HDR/LDRpanoramic images from the LDR segmented images and code them, whereasthe client may receive the coded streams of panoramic images from theserver and decode and display them.

The “panoramic image” as used in this patent specification is notlimited to a “panorama” image in the narrow sense, namely a horizontallyor vertically long image or a panoramic view of 360 degrees, but simplyan image that covers a wide range of area. Also, explained in theembodiments is an example where a panoramic image is generated as asynthesized or composite image. Note that the synthesized image to beoutputted is not necessarily a so-called panoramic image and that thepresent invention is applicable to a normal image of arbitrary sizeserving as the synthesized image. Or the synthesized image to beoutputted may be an image where a plurality of images having differentresolutions are hierarchized. Such a hierarchized image may beconstructed such that when a partial region of image is enlarged, theenlarged region is replaced by an image of a higher resolution.

What is claimed is:
 1. An image generation apparatus comprising: astorage configured to store a plurality of low-dynamic-range (LDR) imagesets each including a plurality of low-dynamic-range (LDR) segmentedimages of different pan angles or tilt angles, each LDR image set beingof a different exposure level, respectively; a high-dynamic-range (HDR)synthesizing unit configured to combine a plurality of LDR segmentedimages of different exposure levels so as to generate ahigh-dynamic-range (HDR) segmented image set including a plurality ofhigh-dynamic-range (HDR) segmented images of different pan angles ortilt angles; and an output image synthesizing unit configured to outputan HDR output image by stitching together adjacent HDR segmented images.2. An image generation apparatus according to claim 1, furthercomprising a detector configured to detect a control point associatingadjacent HDR segmented images by extracting a feature point betweenadjacent HDR segmented images out of the plurality of HDR segmentedimages of different pan angles or tilt angles, wherein the output imagesynthesizing unit generates the HDR output image by adjusting alignmentof and combining the adjacent HDR segmented images using the controlpoint detected by the detector.
 3. An image generation apparatusaccording to claim 1, further comprising a tone mapping unit configuredto compress dynamic range of the plurality of HDR segmented images ofdifferent pan angles or tilt angles by tone mapping so as to generate aplurality of compressed dynamic range segmented images of different panangles or tilt angles, wherein, when a low-dynamic-range (LDR) outputimage is to be generated, the output image synthesizing unit generatesthe LDR output image by stitching together adjacent segmented images outof the plurality of compressed dynamic range segmented images ofdifferent pan angles or tilt angles generated by the tone mapping unit.4. An image generation apparatus according to claim 1, furthercomprising an image coding unit configured to compress and code, throughmotion compensation, an extended output image, when the output image isa moving image, wherein the extended output image is generated such thatpart of a region at a right end of the output image generated by theoutput image synthesizing unit is added to a left end thereof and suchthat part of a region at the left end thereof is added to the right endthereof.
 5. An image display apparatus according to claim 1, wherein theoutput image is a panoramic image.
 6. An image generation methodcomprising: reading out, by a processor, a plurality oflow-dynamic-range (LDR) image sets from a storage device for storing aplurality of LDR image sets each including a plurality oflow-dynamic-range (LDR) segmented images of different pan angles or tiltangles, each LDR image set being of a different exposure level,respectively; combining, by a processor, a plurality of LDR segmentedimages of different exposure levels so as to generate ahigh-dynamic-range (HDR) segmented image set including a plurality ofHDR segmented images of different pan angles or tilt angles; andoutputting, by a processor, an HDR output image by stitching togetheradjacent high dynamic range segmented images.
 7. A program embedded in anon-transitory computer-readable medium, the program comprising: areading module operative to read out a plurality of low-dynamic-range(LDR) image sets from a storage device for storing a plurality of LDRimage sets each including a plurality of low-dynamic-range (LDR)segmented images of different pan angles or tilt angles, each LDR imageset being of a different exposure level, respectively; a high dynamicrange synthesizing module operative to combine a plurality of LDRsegmented images of different exposure levels so as to generate ahigh-dynamic-range (HDR) segmented image set including a plurality ofhigh-dynamic-range (HDR) segmented images of different pan angles ortilt angles; a tone mapping module operative to compress dynamic rangeof the plurality of HDR segmented images of different pan angles or tiltangles by tone mapping so as to generate a plurality of compresseddynamic range segmented images of different pan angles or tilt angles;and an output image synthesizing module operative to generate an HDRoutput image by stitching together adjacent HDR segmented images, whenthe HDR output image is to be generated, and operative to generate anLDR output image by stitching together adjacent segmented images out ofthe plurality of compressed dynamic range segmented images of differentpan angles or tilt angles generated by the tone mapping, when the LDRoutput image is to be generated.
 8. A non-transitory computer-readablemedium encoded with a program, executable by a computer, according toclaim 7.